Make source_semantic_models property accessible from a DataflowPlanNode #1218
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced May 16, 2024
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
courtneyholcomb
approved these changes
May 16, 2024
metricflow/dataflow/dataflow_plan.py
Outdated
| inside of each node, we make those properties of the DataflowPlan, and this node-level converter makes | ||
| such properties easily accessible. | ||
| """ | ||
| return DataflowPlan(sink_nodes=(self,)) |
metricflow/dataflow/dataflow_plan.py
Outdated
| @@ -188,3 +208,24 @@ def __init__(self, sink_nodes: Sequence[DataflowPlanNode], plan_id: Optional[Dag | |||
| @property | |||
| def sink_node(self) -> DataflowPlanNode: # noqa: D102 | |||
| return self._sink_nodes[0] | |||
|
|
|||
| def __complete_subgraph(self, node: DataflowPlanNode) -> Sequence[DataflowPlanNode]: | |||
There was a problem hiding this comment.
Should this be a staticmethod?
There was a problem hiding this comment.
Sure, I can do that.
tlento
force-pushed
the
make-source-semantic-models-available-from-nodes
branch
from
c1faf9a to
909aabb
Compare
tlento
force-pushed
the
use-pushdown-params-for-disabling-time-constraint-pushdown
branch
from
dc94cfc to
31ee2df
Compare
tlento
force-pushed
the
make-source-semantic-models-available-from-nodes
branch
from
909aabb to
9ef3244
Compare
This was referenced May 22, 2024
tlento
force-pushed
the
use-pushdown-params-for-disabling-time-constraint-pushdown
branch
from
31ee2df to
3a02ea7
Compare
tlento
force-pushed
the
make-source-semantic-models-available-from-nodes
branch
from
9ef3244 to
45cf01f
Compare
tlento
force-pushed
the
use-pushdown-params-for-disabling-time-constraint-pushdown
branch
from
3a02ea7 to
9769994
Compare
Base automatically changed from
use-pushdown-params-for-disabling-time-constraint-pushdown
to
main
May 24, 2024 00:46
This change ultimately adds the source_semantic_models property to the DataflowPlan and adds a hook for enabling access to it from any arbitrary DataflowPlanNode. We currently have two use-cases for this, one in the cloud codebase that needs the semantic model inputs for a dataflow plan, and the upcoming predicate pushdown evaluation which needs the semantic model inputs for a given DataflowPlanNode. An earlier version of this change added the property directly to the DataflowPlanNode, which would satisfy both use cases above. The issue with having this property assigned directly to a DataflowPlanNode is that the property might be considered both a node-level and graph-level attribute, so it's not clear where to put the accessor. The solution we came up with for this was to allow access to a DataflowPlan DAG object built from the node, which would effectively encapsulate the subgraph represented by the node and its ancestors. Then we can access these subgraph properties through the DataflowPlan while making it clear to the caller that what they are asking for is a subgraph-level, rather than a node-level attribute.
tlento
force-pushed
the
make-source-semantic-models-available-from-nodes
branch
from
45cf01f to
f0c6cb9
Compare
This was referenced Jun 12, 2024
This was referenced Jun 25, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change ultimately adds the source_semantic_models property to
the DataflowPlan and adds a hook for enabling access to it from any
arbitrary DataflowPlanNode.
We currently have two use-cases for this, one in the cloud codebase
that needs the semantic model inputs for a dataflow plan, and the
upcoming predicate pushdown evaluation which needs the semantic model
inputs for a given DataflowPlanNode.
An earlier version of this change added the property directly to the
DataflowPlanNode, which would satisfy both use cases above. The issue
with having this property assigned directly to a DataflowPlanNode
is that the property might be considered both a node-level and graph-level
attribute, so it's not clear where to put the accessor.
The solution we came up with for this was to allow access to a DataflowPlan
DAG object built from the node, which would effectively encapsulate the
subgraph represented by the node and its ancestors. Then we can access
these subgraph properties through the DataflowPlan while making it clear
to the caller that what they are asking for is a subgraph-level, rather
than a node-level attribute.